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ThinkerTools 

No studies of ThinkerTools that fall within the scope of the Science review protocol meet What Works 
Clearinghouse (WWC) evidence standards. The lack of studies meeting WWC evidence standards 
means that, at this time, the WWC is unable to draw any conclusions based on research about the 
effectiveness or ineffectiveness of ThinkerTools on middle school students. Additional research is 
needed to determine the effectiveness or ineffectiveness of this intervention. 

Program Description^ 

ThinkerTools^ is a computer-based program that aims to deveiop students’ understanding of physics and scientific 
modeiing. The program is composed of two curricula for middle school students, ThinkerTools Inquiry and Model- 
Enhanced ThinkerTools. ThinkerTools Inquiry allows students to explore the physics of motion and then asks them 
to apply that knowledge to solve real-world problems. In the Model-Enhanced ThinkerTools curriculum, students 
create computer models that express their own theories of force and motion. 

During the program, students navigate through four environments of increasing complexity known as microworlds: 
motion in one dimension, motion in two dimensions, understanding continuous forces, and analyzing trajectories. 
Students can run simulations of moving objects and observe the effects of various forces such as impulse, gravity, 
and friction. Measurement tools enable students to calculate distance, time, and velocity. The social Interaction in 
the ThinkerTools classroom is similar to that of an actual scientific community, using a cycle of inquiry. Research 
begins with a whole-class forum and is carried out in groups. These groups make predictions, do experiments, and 
share their findings. 

Research^ 

The WWC identified four studies of ThinkerTools for middle school students that were published or released 
between 1 990 and 201 1 . 

• Two studies are within the scope of the Science review protocol but do not meet WWC evidence standards. 
These quasi-experimental studies did not establish that the comparison group was comparable to the treatment 
group prior to the start of the intervention. 

• Two studies are out of the scope of the Science review protocol because they have an ineligible study design. 
For the purposes of this report, the studies did not use a valid comparison group, as authors compared Model- 
Enhanced ThinkerTools to the prior curriculum version, ThinkerTools Inquiry. 
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Endnotes 

■' The descriptive information for this program was obtained from pubiiciy avaiiabie sources: the program’s website (http://thinkertools.org/, 
downioaded October 201 1) and White (1993). The WWC requests deveiopers to review the program description sections for accu- 
racy from their perspective. The program description was provided to the deveioper in October 201 1 ; however the WWC received 
no response. Further verification of the accuracy of the descriptive information for this program is beyond the scope of this review. 
The iiterature search refiects documents pubiiciy avaiiabie by June 2011. 

^ This review focuses on ThinkerTools Inquiry and Model-Enhanced ThinkerTools, but it uses the generai term ThinkerTools to encom- 
pass both programs. 

® The studies in this report were reviewed using WWC Evidence Standards, Version 2.1 , as described in the review protocol for the 
Science topic area. The evidence presented in this report is based on avaiiabie research. Findings and conciusions may change as 
new research becomes avaiiabie. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2012, February). 
Science intervention report: ThinkerToois. Retrieved from http://whatworks.ed.gov. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 

Single-case design 
Standard deviation 



Statistical significance 



Substantively important 



Attrition occurs when an outcome variable is not avaiiabie for aii participants initiaiiy assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If treatment assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent of 
evidence levels are given in the WWC Procedures and Standards Handbook (version 2.1). 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to treatment and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into treatment and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the research 
design and the magnitude, statistical significance, and consistency in findings. The criteria for the 
ratings of effectiveness are given in the WWC Procedures and Standards Handbook (version 2.1). 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 



Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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